Brightness correction of a pixel array in an image sensor

ABSTRACT

An image processor is provided for correcting brightness of saturated pixels of a captured image. The image processor can include a pixel saturation determiner that whether one or more pixels in an image sensor have been saturated by comparing pixel brightness levels of the pixels to a predetermined saturation threshold. Moreover, the image processor includes an image enhancer that generates a corrected image without artifacts due to the saturated pixel(s) by replacing the pixel brightness of the saturated pixel(s) with a pixel correction value that is based on a pixel brightness of one or more unsaturated pixel in the image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

The current application claims priority to U.S. Patent ProvisionalApplication No. 62/385,153, filed Sep. 8, 2016, the entire contents ofwhich are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to image data and moreparticularly, to bright correction in image data.

BACKGROUND

When using, for example, a camera, to take pictures, bright areas of ascene may saturate one or more pixels on a sensor in the bright area ofthe picture. The pixels may not all saturate at the same brightnesslevel. In a bright area, one pixel may saturate before other pixelsaround the saturated pixel. One pixel saturating before other pixelsaround the saturated pixel may sometimes occur with shared pixels. Ashared pixel is a structure in which a part of a set of pixeltransistors is shared by a plurality of pixels. With the shared pixelstructure, a selection transistor, a reset transistor, and anamplification transistor may be shared, for example. The selectiontransistor, the reset transistor, and the amplification transistor maybe shared in a plurality of pixels.

By using a shared pixel structure, an area of the pixel transistors,e.g., the selection transistor, the reset transistor, and theamplification transistor, may be arranged such that an area of eachpixel may be decreased. For a shared pixel, there may be differences insaturation level between the pixels of the shared pixel structure. Oneexample of a shared pixel structure that may include pixels havingdifferent saturation levels is the quincunx pixel structure. Anotherexample shared pixel structure that may include pixels having differentsaturation levels is a quadruplet pixel structure.

Other structures may include pixels having different saturation levels.Furthermore, other non-shared pixel structures may also include pixelshaving different saturation levels. For example, a quadruplet pixelstructure that is not also a shared pixel structure may include pixelshaving different saturation levels.

SUMMARY

The following presents a simplified summary of one or more aspects inorder to provide a basic understanding of such aspects. This summary isnot an extensive overview of all contemplated aspects, and is intendedto neither identify key or critical elements of all aspects nordelineate the scope of any or all aspects. Its sole purpose is topresent some concepts of one or more aspects in a simplified form as aprelude to the more detailed description that is presented later.

An aspect may include a camera sensor system. The camera sensor systemmay include an image sensor. Additionally, the camera sensor system mayinclude a processor coupled to the camera sensor. The processor may beconfigured to process an image from the image sensor by comparing apixel brightness of the image to a first threshold and a secondthreshold. The processor may also be configured to select the pixelbrightness when the pixel brightness is below the first threshold.Additionally, the processor may be configured to select a neighboringpixel brightness when the pixel is above the second threshold. Theprocessor may also be configured to perform an average of pixelbrightness when the pixel brightness is between the first and secondthreshold.

According to another exemplary aspect, an image processing system isprovided for correcting brightness of saturated pixels in an imagesensor. In this aspect, the image processing system includes a pixelarray comprising a plurality of pixels each configured to capture imagedata; a pixel brightness measurement engine configured to measure apixel brightness of the at least one pixel of the plurality of pixelsfor the captured image data; a pixel saturation determiner configured todetermine that the at least one pixel is saturated when the measuredpixel brightness is greater than a predetermined pixel saturationthreshold; a pixel brightness interpolator configured to calculate aninterpolated brightness value to correct the saturated at least onepixel based on the measured brightness value of the saturated at leastone pixel and a brightness value of at least one neighboring pixel ofthe plurality of pixels that is adjacent to the saturated at least onepixel; a pixel brightness corrector configured to correct the pixelbrightness of the saturated at least one pixel based on the calculatedinterpolated brightness value; and an image generator configured todisplay on a display device the captured image data having the correctedpixel brightness of the saturated at least one pixel.

In another aspect, an image processing system is provided for correctingbrightness of saturated pixels in an image sensor. In this aspect, theimage processing system includes a pixel brightness measurement engineconfigured to measure a pixel brightness of a plurality of pixels forimage data captured by a camera; a pixel saturation comparatorconfigured to compare a measured pixel brightness of at least one of theplurality of pixels with first and second saturation thresholds; a pixelbrightness selector configured to select the measured pixel brightnessof the at least one pixel as an output pixel brightness when themeasured pixel brightness of the at least pixel is less than the firstsaturation threshold, and to select a measured pixel brightness of atleast one neighboring pixel as the output pixel brightness when themeasured pixel brightness of the at least pixel is greater than thesecond saturation threshold; and an image generator configured todisplay on a display device the image data having the selected outputpixel brightness of the at least one pixel of at least one of theplurality of pixels.

In yet another aspect, an image processor for correcting brightness ofsaturated pixels of a captured image. In this aspect, the imageprocessor includes a pixel saturation determiner configured to determinewhether at least one pixel in an image sensor is saturated when a pixelbrightness of the at least one pixel is greater than at least onesaturation threshold; and an image enhancer configured to generate acorrected image without one or more image artifacts due to the saturatedat least one pixel by replacing the pixel brightness of the saturated atleast one pixel with a pixel correction value based on a pixelbrightness of at least one unsaturated pixel in the image sensor.

To the accomplishment of the foregoing and related ends, the one or moreaspects comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more aspects. These features are indicative, however, ofbut a few of the various ways in which the principles of various aspectsmay be employed, and this description is intended to include all suchaspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example camera system 100 having aviewfinder display according to an exemplary embodiment.

FIG. 2A is block diagram of an example detail correction circuit of apixel processor enhanced by using an intermediate exposure for thedetail correction signal according to an exemplary embodiment.

FIG. 2B is a block diagram of an example subtractor for determiningmultiple intermediate exposures according to an exemplary embodiment.

FIG. 3 a graphical illustration of example pixel charge rates during asequence of exposures as an indication of motion presence.

FIG. 4 is a diagram illustrating a quadruplet pixel structure.

FIG. 5 is a diagram illustrating example gains for the four examplepixels of the quadruplet structure of FIG. 4.

FIG. 6 is a diagram illustrating an example of a pattern that may formdue to differences in gain for various pixels in an image sensor.

FIG. 7 is a gamma chart illustrating gray value along the y-axis anddistance along the x-axis.

FIG. 8 is a diagram illustrating an example set of pixels and a graphillustrating gains that may be used to fade to a quincunx interpolationfor the bright areas to avoid artifacts according to an exemplaryembodiment.

FIG. 9 is an illustration of two pixels interpolated each based on a setof two neighboring pixels according to an exemplary embodiment.

FIG. 10 is an example graph illustrating gain along the y-axis andluminance along the x-axis according to an exemplary embodiment.

FIG. 11 is a gamma chart illustrating gray value along the y-axis anddistance along the x-axis according to an exemplary embodiment.

FIG. 12 illustrates an uncorrected pattern on an example picture.

FIG. 13 illustrates of a result for a corrected pattern on an examplepicture according to an exemplary embodiment.

FIG. 14 illustrates an example graph for uncorrected and corrected leastsignificant bits according to an exemplary embodiment.

FIG. 15 is an example block diagram illustrating a system for pixelbrightness correction according to an exemplary embodiment.

FIG. 16 is a flowchart illustrating an example method for pixelbrightness correction according to an exemplary embodiment.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of various configurations and isnot intended to represent the only configurations in which the conceptsdescribed herein may be practiced. The detailed description includesspecific details for the purpose of providing a thorough understandingof various concepts. However, it will be apparent to those skilled inthe art that these concepts may be practiced without these specificdetails. In some instances, well-known structures and components areshown in block diagram form in order to avoid obscuring such concepts.

Certain aspects of video production systems will now be presented withreference to various apparatus and methods. These apparatus and methodswill be described in the following detailed description and illustratedin the accompanying drawing by various blocks, modules, components,circuits, steps, processes, algorithms, etc. (collectively referred toas “elements”). These elements may be implemented using electronichardware, computer software, or any combination thereof. Whether suchelements are implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem.

By way of example, an element, or any portion of an element, or anycombination of elements may be implemented with a “pixel processor” thatincludes one or more processors. Examples of processors includemicroprocessors, microcontrollers, image processors, digital signalprocessors (DSPs), field programmable gate arrays (FPGAs), applicationspecific integrated circuits (ASICs), programmable logic devices (PLDs),state machines, gated logic, discrete hardware circuits, and othersuitable hardware configured to perform the various functionalitiesdescribed throughout this disclosure. One or more processors in theprocessing system may execute software. Software shall be construedbroadly to mean instructions, instruction sets, code, code segments,program code, programs, subprograms, software modules, applications,software applications, software packages, routines, subroutines,objects, executables, threads of execution, procedures, functions, etc.,whether referred to as software, firmware, middleware, microcode,hardware description language, or otherwise. The software may reside ona non-transitory computer-readable medium. A computer-readable mediummay include, by way of example, non-transitory storage such as amagnetic storage device (e.g., hard disk, floppy disk, magnetic strip),an optical disk (e.g., compact disk (CD), digital versatile disk (DVD)),a smart card, a flash memory device (e.g., card, stick, key drive),random access memory (RAM), read only memory (ROM), programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), aregister, a removable disk, as well as a carrier wave, a transmissionline, and any other suitable medium for storing or transmittingsoftware. The computer-readable medium may be resident in the processingsystem, external to the processing system, or distributed acrossmultiple entities including the processing system. Those skilled in theart will recognize how best to implement the described functionalitypresented throughout this disclosure depending on the particularapplication and the overall design constraints imposed on the overallsystem.

FIG. 1 is a simplified block diagram illustrating an example camerasystem 100 that implements a pixel processor 122. Camera system 100 maycomprise a camera 102, the viewfinder 104, and a lens system 106. Camera102 may include an image sensor 120, which may comprise an array ofpixels to convert photons to electrical charges. Among others, the imagesensor may comprise a charged coupled device (CCD) or complementarymetal oxide semiconductor (CMOS). Thus, the array of pixels may comprisean array of CCD or CMOS pixels. An image is projected by the lens system106 onto the image sensor 120. The output of the image sensor 120comprises an output signal from the array of pixels. The image sensor120 produces a voltage signal by converting the photon input level foreach pixel to a proportional voltage signal for each pixel in the array.The pixels of the image sensor 120 are exposed to photons, e.g.,projected by the lens system 106, and read by the pixels of the imagesensor in time units of frame exposures. Each pixel accumulates anelectrical charge representative of the image during the course of theexposure frame. The image sensor 120 may convert the electrical chargeto an analog output voltage signal. Alternatively, the image sensor 120may convert the electrical charge to an analog voltage and convert theanalog voltage to a digital signal using an analog to digital converter,for example, to produce a digital output voltage signal. The imagesensor 120 may transmit the output voltage signal periodically at theframe rate. The pixel may be reset by discharging the accumulated chargeso that the next pixel charge accumulation for the next frame can begin.The amount of light photons may be converted to the voltage signal up toa saturation threshold, at which point no further charge can beaccumulated for the pixel output. In one example, multiple image sensors120 may operate in a synchronous manner. Multiple image sensors 120 mayalso operate in different phases relative to one another.

Pixel processor 121 may be configured to correct the image sensor pixeloutput signals for motion blur. The output of the pixel processor 121may be an array of pixel signals to form an image for each frameexposure of the video sequence. Camera 102 includes a video processor122 that receives a sequence of images and produces a digital videooutput having a desired frame rate, aspect ratio, etc. The videoprocessor 122 may also perform white balance, color correction, andgamma correction to the video images. The video processor 122 may beimplemented as a plurality of separate processors each configured toperform one or more of the above functions. Alternatively, the pixelprocessor 121 and video processor 122 may be arranged in a reversemanner, whereby the pixel processor 121 processes images on a per pixelbasis already corrected by the video processor 122.

An encoder 124 may receive a raw video output from video processor 122and produce a formatted digital video signal encoded according to aparticular specification (e.g., Serial Digital Interface (SDI),H.264/MPEG-4 Advanced Video Coding, or High Definition MultimediaInterface (HDMI)). The signal from encoder 124 may be output fortransmission to a video production system and/or over a network usingtransceiver 126. Encoder 124 may also provide an encoded or raw videofeed to viewfinder 104.

Viewfinder 104 may include a decoder 141 configured to receive encodedvideo or raw video from encoder 124 and provide image data for thedisplay 142. In one example, the display 142 may include an organiclight-emitting diode (OLED) at each pixel, whereby a light-emittingdiode (LED) is coated with an emissive electroluminescent layer formedfrom an organic compound which emits light in response to an electriccurrent. These and other devices may be used to generate images on thedisplay 142.

Lens system 106 may include one or more lenses and may be controlled toprovide a desired optical configuration of lenses, which configurationmay specify, for example, a depth of field setting, a numericalaperture, and a focal length.

FIG. 2A is block diagram of an example detail correction circuit 200 ofthe pixel processor 121 for sampling the image sensor 120 during anintermediate portion of the frame to produce a signal representative ofthe image. For each pixel of the image sensor 120, the sampler 202receives input 201, which is the pixel output received from the imagesensor 120. The sampler 202 may read the cumulative pixel output values(e.g., an electrical charge or output voltage signal values) at discretetime samples S1 to Sn. Samples S1 and Sn may occur at the beginning andend of an exposure frame, respectively, with one or more samplesoccurring between samples S1 and Sn. For example, with n=4, sample Sn=S4occurs at the end of the frame exposure, a sample S2 may occur at about25% of the full frame exposure interval, and a sample S3 may occur atabout 75% of the full frame exposure interval. Alternatively, sample S2may occur at about 33% of the full frame exposure interval, and a sampleS3 may occur at about 66% of the full frame exposure interval. As analternative example, with n=3, sample S3 occurs at the end of theexposure frame, and a sample S2 may occur anywhere between about 25 to75% of the frame exposure. Other possible alternative variations for n>4may be implemented, where sampler 202 provides additional samples ofpixel output values within the full frame exposure. For a CMOSimplementation of image sensor 120, the sampler 202 may operate asdescribed above to read the multiple samples per exposure frame. In thecase of a CCD implementation of image sensor 120, reading intermediatesamples may not be possible within a single frame. Consequently, a CCDimage sensor 120 may need to operate at a faster frame rate than thenominal rate for the camera 100 to simulate multiple samples within thenominal frame. For example, if the video signal for camera 100 isprocessed at rate of 50 FPS (20 ms frames), and sampler 202 requiresfour samples per frame (n=4), then the CCD image sensor 120 may operateat a faster rate of 200 FPS (5 ms frames), yielding 4 CCD images per 20ms frame. Each set of samples S1 to S4 may then be derived from a blockof four CCD images for an equivalent 20 ms frame.

A subtractor 203 determines the cumulative pixel output (e.g.,electrical charge or output voltage signal value) for the full frameexposure by subtracting the pixel output value at sample S1 from thecumulative pixel output value at sample Sn. A subtractor 204 determinesthe cumulative pixel output value of an intermediate frame exposure bysubtracting the cumulative pixel output value at sample S2 fromcumulative pixel output value at sample S3. For the example of n=4,sample S2 at 25% of the exposure frame and sample S3 at 75% of the fullframe exposure interval, the intermediate exposure provides the pixeloutput value for the middle 50% of the frame exposure. For the examplewhere n=4, sample S2 occurring at about 33% of the frame exposure andsample S3 at about 66% of the full frame exposure, the intermediateexposure provides the pixel output for the middle third of the fullframe exposure. Alternatively, for the example of n=3, sample S3 occursat the end of the full frame exposure, and sample S2 at 50% of the fullframe exposure, subtractor 204 may subtract the pixel output value atsample S2 from the pixel output value at sample S3 to provide anintermediate exposure value related to the last half of the full frameexposure. Alternatively, subtractor 204 may subtract the pixel outputvalue at sample S1 from the pixel output value at sample S2 to providean intermediate exposure value related to the first half of the fullframe exposure.

FIG. 2B shows a block diagram for an example subtractor 204, in whichmultiple intermediate exposures may be determined. In an embodiment withsampler 202 reading samples Sn for n>4, multiple intermediate exposuresmay be obtained by subtractor 204 by using multiple subtractors 214 toeach determine the respective intermediate exposure separately. Forexample, for n=6, intermediate exposures S6-S5, S5-S4, S4-S3, S3-S2 andS2-S1 may be determined by one or more subtractors 214. Selector 224 maybe implemented as a multiplexer to adaptively select which intermediateexposure is to be processed by the detail correction circuit 200.

Amplifier 205 receives the pixel output of the intermediate frameexposure and amplifies it as a normalization to a full frame exposure.For example, the amplifier 205 may apply a 6 dB boost to the cumulativepixel output value. A detail processor 206 receives the amplified pixeloutput value and performs a detail correction algorithm to correctmotion blur. The detail processor 206 improves the perceived imagesharpness by generating a correction signal at any signal transition.Transitions in luminance and/or chrominance are emphasized by the detailprocessor 206 to enhance objects in a scene. The calculated detailcorrection is added to the original image on a pixel-by-pixel basis.This detail correction signal depends on the sharpness of the image. Inthis example, the intermediate exposure contains 50% of the motion blurfor the pixel. By performing detail correction on the shorterintermediate frame exposure instead of the full frame exposure, theeffect of motion blur in the pixel is reduced, which enhances theeffectiveness of the detail correction. Summer 207 is configured to addthe detail correction signal to the full exposure pixel output, givingan enhanced pixel output 208.

FIG. 3 is a graphical illustration of an example sampling of an imagesensor 120 pixel output for multiple samples S1 to Sn during a series ofsingle frame exposures 301, 302, 303. In this example, n=4 andcumulative pixel output values are sequentially read at samples S1, S2,S3 and S4, with samples S1 and S4 providing the cumulative pixel outputfor a full frame exposure as sample S1 occurs at the beginning of theframe, and sample S4 occurs at the end of the frame. A pixel output ofimage sensor 120 for an intermediate frame exposure can be obtainedbetween samples S2 and S3. For exposure 301, the constant slope of theaccumulating pixel output during the interval between S1 and S4indicates a constant photon input for this pixel, which means a constantlight level is being reflected from an object in the camera field ofview. Since the light level is constant, the object is likely fixed andnot moving across this pixel unit within the frame exposure 301. Incontrast, the frame exposures 302 and 303 illustrate pixel charge ratevariation between each sample pair interval (e.g., Si and Si+1). Forexample, in exposure 302, the rate of pixel charge accumulation isconstant between samples S1 and S2, S2 and S3, but falls between samplesS3 and Sn, revealing the presence of motion for the pixel output, from abrighter object to a darker object, and hence the potential for motionblur. Exposure 303 shows the presence of motion as a transition from adarker object to a brighter object being sensed by the pixel of imagesensor 120 (i.e., more light photons being sensed by the pixel), as thepixel charge rate increases between samples S3 and Sn compared to thepixel charge rate between S1 and S3. Thus, with the sampler 202configured to track intermediate exposures within each frame, (e.g.,between samples S2 and S3), motion is detectable. In contrast, aconventional approach which measures pixel output only at the beginningof the frame and at the end of the frame (i.e., at samples S1 and Sn)would give misleading pixel charge rates 312 and 313, and overlook theindication of motion.

FIG. 4 is a diagram illustrating a quadruplet pixel structure 400. Thequadruplet pixel structure may have four sub-pixels, e.g., 1, 2, 3, 4.The quadruplet pixel structure may be made up of four individualsub-pixels. In another example, the four sub-pixels, 1, 2, 3, 4, may befour sub-pixels of a shared pixel structure.

Each sub-pixel in the example of FIG. 4, may have a different gain.Accordingly, each of the sub-pixels may saturate at a different level.For example, for a given brightness, e.g., near saturation, some of thefour sub-pixels may saturate, while some of the four sub-pixels may notsaturate. For example, pixels with higher gains may have more of atendency to saturate because the higher gain may drive the pixels valueshigher for a given brightness.

FIG. 5 is a diagram illustrating example gains for the four examplepixels of the quadruplet structure 400 of FIG. 4. In general, a pixel'sgain is how the amount of signal given by the pixel varies as a functionof the amount of input light (or equivalent), i.e., accumulated chargeduring image capture by the image sensor having the pixel array. Thegains of the sub-pixels 1, 2, 3, 4, may correspond to the gains Q1, Q2,Q3, Q4 illustrated in FIG. 5. As illustrated in FIG. 5, the gains for Q1and Q4, for example, may be higher than the gains for Q2 and Q3.Accordingly, the sub-pixels 1 and 4 may saturate more easily than thesub-pixels 2 and 3.

In one example, the separate gains for Q1, Q2, Q3, Q4 may be correctedusing an average. For example, pixel 1, gain Q1, may be corrected as:Q1/((Q1+Q2+Q3+Q4)/4). In other examples, a high gain may be disregarded.For example, if the gain for pixel 1, Q1 is greater than the gain forthe other pixels, it may be disregarded and pixel 1 may be corrected asthe average of the other pixels, e.g., Q2+Q3+Q4/3. In some examples,each quarter of a set of pixels may be averaged over a large area, forexample, greater than 50% of an image area may be averaged.

In an example, gain may be calculated using digital double sampling(DDS) values, which are measured electrical values with removedundesired offsets. Because black is at 0 least significant bit (LSB),only a gain is needed, without an offset value. In an example, the rangeto be corrected is ˜2-5%. A large area photo response non-uniformity(PRNU) specification is 0.06%. In an example, 2000*LSB14*0.06%=1,2LSB14. The gain control range therefore should be ufix(0,16,15).

In some examples, a camera may include a white calibration, which shouldbe available in service menu. White calibration should run duringproduction on a predefined white scene. The result of the whitecalibration should be stored in memory on a sensor board. The storedresults of the white calibration may be loaded during startup in camera,e.g., in less than 10 ms in some examples. Additionally, the camerashould be able to switch the use of the white calibration data off incase of emergencies such as data load errors, such as data load errorsof the results of the white calibration. In FIG. 5 gain is along they-axis. Luminance values are along the x-axis. As illustrated in FIG. 5,the gain may not be flat for the whole range of luminance values.

FIG. 6 is a diagram illustrating an example of a pattern that may formdue to differences in gain for various pixels in an image sensor. Asdiscussed, the gain is not flat for the whole range. 100% level (Green)is at 1500 LSB14 (F4 sensitivity). The range from 0-3000 may includeLSB14 that is relatively flat. For high illuminations (e.g., >300%, apattern becomes visible.

FIG. 7 is a gamma chart illustrating gray value along the y-axis anddistance along the x-axis. For distance values, in pixels, belowapproximately 2500, the gray values are fairly linear. At high distancevalues, however, variations increases. A pattern exists on high levels.As illustrated in FIG. 7, before correction, there is a differentsaturation level per quadruplet. Note that a pattern forms, asillustrated in FIG. 7.

FIG. 8 is a diagram illustrating an example set of pixels and a graphillustrating gains that may be used to fade to a quincunx interpolationfor the bright areas to avoid artefacts. The graph includes a normalizedgain on the y-axis and a luminance value on the x-axis. In an example,brightness values for pixels near saturation may be used to interpolatevalues for pixels that are saturated. One example may fade to a quincunxinterpolation for the bright areas to avoid artefacts.

For example, when a brightness value for a pixel is below a threshold(thlow), the brightness value for that pixel may be used. When thebrightness value for a pixel is above a threshold (thhigh) thebrightness value for a neighboring pixel or a group of neighboringpixels may be used. Between the thresholds (thlow) and (thhigh) a fadebetween pixels may be used, as illustrated in FIG. 8. The pixelsaturation thresholds (e.g., saturation thresholds (thlow) and (thhigh))can be predetermined and stored in memory of camera 102 and accessed bythe pixel processor 121 during image processing according to anexemplary aspect.

Accordingly, as illustrated in the graph of FIG. 8, below the threshold(thlow), the gain used 802 may be “1.” Thus, when a brightness value fora pixel is below a threshold (thlow), the brightness value for thatpixel may be used because the gain for that pixel is “1.” When thebrightness value for a pixel is above a threshold (thhigh) thebrightness value 804 for a neighboring pixel or a group of neighboringpixels may be used because the gain for the neighboring pixel or groupof pixels is “1,” while the gain for that pixel is “0.” Between thethresholds (thlow) and (thhigh) a fade between pixels may be used, asillustrated in FIG. 8, because the gain for that pixel is transitioningfrom “1” to “0,” while the gain for the neighboring pixel or group ofpixels is transitioning from “0” to “1.”

When pixels are saturated, the brightness values for the pixels may bedetermined based on neighboring pixels. Accordingly, as illustrated inFIG. 8, values for a series of saturated pixels may be interpolated oraveraged to neighboring brightness values. The colors in the chartindicate different pixels of interest, rather than actual colors. Thelighter pixels on the left image may be pixels for which brightnessvalues are being determined based on the systems and methods describedherein. The darker pixels on the left image may be pixels used todetermine brightness values for the lighter pixels in conjunction withthe starting brightness values for the lighter pixels themselves. Asdiscussed above, below a threshold (thlow), the values for the lighterpixels may be used since the pixels are not saturated. Above a threshold(thhigh), the lighter pixels are determined to be saturated socorresponding brightness values of the unsaturated darker pixels may beused. Between the thresholds, a fade may be used as further describedherein.

For an example where the lighter pixels are all above a thresholds(thhigh), in some cases, the lighter color indicates pixels having asaturated brightness. In the chart the darker pixels may be interpolatedtowards the brightness values for the darker as indicated by the“corrected pixels” after processing. Again, the colors designate certainsets pixels rather than colors for those pixels.

FIG. 9 is an illustration of two pixels interpolated each based on a setof two neighboring pixels. In FIG. 9 each interpolated pixel isinterpolated towards a brightness value for the two neighboring pixels.In other words, the two outer pixels are neighboring, unsaturated pixelsand the middle pixels have been saturated, i.e., the interpolated pixelsmay be saturated. Accordingly, values for the neighboring pixels may beused. As illustrated in FIG. 9, the middle pixels are interpolated. Thethreshold, Thlow is the starting point where original pixel is used. Thethreshold Thhigh is the minimum, beyond which the pixel is fullyinterpolated. The systems and methods described herein may beimplemented with a threshold and a slope setting as indicated by thefollowing:

%Quadruplet fader threshold = 4000; %LSB14 slope = 10; % Gain on faderfor i = 1 : 2{circumflex over ( )}14  Vin(i) = i;  quincunxfadergain(i)= max( min( ((i-threshold)*slope) + 2{circumflex over ( )}13 ,2{circumflex over ( )}14),0); end plot(Vin, quincunxfadergain ); holdon; plot(Vin, 2{circumflex over ( )}14-quincunxfadergain ); hold off;grid on;

FIG. 10 is an example graph illustrating gain along the y-axis andluminance along the x-axis. Unlike the graph illustrated in FIG. 8, thegraph illustrated in FIG. 10 is not normalized. Accordingly, the graphillustrated in FIG. 10 includes possible gain and luminance values in anexample system. Similar to the discussion with respect to FIG. 8, asillustrated in FIG. 10, when the pixels above the threshold, they arereplaced with an interpolated value. Pixels below a threshold are used,while pixels between the thresholds are combined or faded together. Notethat the pattern is much more consistent.

FIG. 11 is a gamma chart illustrating gray value along the y-axis anddistance along the x-axis. For distance values, in pixels, belowapproximately 2500, the gray values are fairly linear. Using the brightcorrection described herein, above 2500, the gray values remain fairlylinear. A pattern exists on high levels. As illustrated in FIG. 7,before correction, there is a different saturation level per quadruplet.

In some examples, camera requirements may include putting values in acontrol map with default settings. With a Delphi tool values may bechanged when needed, but, in some examples, the values arenon-adjustable when an example system is being used.

FIG. 12 illustrates an uncorrected pattern on an example picture. Amodulo 4k correction may be similar as in LDK86 High-speed mode, amodulo pattern is visible. In U23A this shows as a modulo 128. Becausean example has 2112 columns with a shared pixel, the correction isproposed to be modulo 2k. This means each gain/offset value can be usedfor 2 consecutive pixels.

In one example, a full 4k resolution correction may be used. Anotherexample may use sensors for ˜2k columns, where each column connects to arow of 1125 four-shared pixels. Using ˜2k columns results in a sensorresolution of 4k×2250. Another example with the 2k columns may use a 2kcorrection, thus, such an example may use every correction value 2x inhorizontal direction. This may mean a reduction of hardware resources of2x while maintaining our correction functionality. For different pixelstructures this 4k/2k choice may change.

In an example, correction may be performed by sum all rows from a DDS-edimage (modDDS), filter this result with at least 128 samples (in case of2k, this reduces to 64) (AVG_DDS), sum all rows from a Raw_Dark image(modDark), and filter this result as the DDS-average line (AVG_Dark).

alpha=(modDDS−AVG_DDS)./AVG_DDS; % Use DDS as source

offset=AVG_Dark−(modDark−(alpha.*modDark));

-   -   Accuracy may be: alpha=ufix(0,16,15), offset=sfix(1,16,4).        Applying the resulting gain alpha and offset:

IM_BrightCorr=IM_bright−(IM_bright.*IM_gain+IM_offset);

IM_Bright2Corr=IM_bright2−(IM_bright2.*IM_gain+IM_offset);

IM_DarkCorr=IM_dark−(IM_dark.*IM_gain+IM_offset);

IM_DDS1_Corr=double(IM_BrightCorr−IM_DarkCorr)/2;

IM_DDS2_Corr=double(IM_Bright2Corr−IM_BrightCorr)/2;

IM_DDSCorr=mask_odd.*IM_DDS1_Corr+(mask_even.*IM_DDS2_Corr)*OddEvenGain;

FIG. 13 illustrates of a result for a corrected pattern on an examplepicture.

FIG. 14 illustrates an example graph for uncorrected and corrected leastsignificant bits. In an example, a same set of camera systemrequirements may be used as with quadruplet correction.

FIG. 15 is an example block diagram illustrating a system for pixelbrightness correction according to an exemplary embodiment. For example,the bright correction may be implemented within the quadrupletcorrection block. The quadruplet correction block may be controlled by aprocessor.

As described herein, in very bright areas a pixel will saturate. In ashared pixel, there might be differences between the shared pixel on thesaturation level. An example is a pixel with quincunx structure, butother structures might have same effect.

In very bright areas resolution may be sacrificed in favor of artifactswhen pixel starts saturating. The subpixel with best performance may bekept. Additionally, the subpixel with best performance may be may beinterpolated to replace the subpixel which is saturating on a lowerillumination level. In an example, a way of mixing from one level toanother may be done with a fader to eliminate fast switching. An examplemay keep a lower illuminated part of a scene untouched, and for highbrightness parts of the scene the deviating pixels may be interpolatedfrom the brightest ones through the quincunx interpolation, as opposedto clipping the higher part and ignore the higher values which startdeviating. In some examples, the usable dynamic range of the imager isextended.

As shown, an image processing system 1500 (or image processor) isprovided for correcting brightness of saturated pixels in an imagesensor. For example, the image sensor can include a pixel array 1502having a plurality of pixels each configured to capture image data.Moreover, a pixel brightness measurement engine 1504 is provided andconfigured to measure a pixel brightness of the pixels in the pixelarray for the captured image data according to the methods describedabove. As further shown, a pixel saturation determination unit 1506 canbe configured to perform DDS to calculate the pixel gains and,therefore, further determine whether the one or more pixel are saturatedif the measured pixel brightness is greater than a predetermined pixelsaturation threshold, as further described above.

In addition, a pixel brightness interpolator 1508, which can be one ormore microprocessors, for example, is calculate interpolated brightnessvalue to correct the saturated pixel based on the measured brightnessvalue of the saturated pixels and a brightness value of neighboringpixels in the pixel array. In a refinement of the exemplary aspect, themicroprocessor 1508 can determine a pixel correction value, which can bedependent on whether the measured pixel brightness value is above, belowor between the thresholds as further described above. Based on thecalculated interpolated brightness value (e.g., a pixel correctionvalue), a pixel brightness corrector 1510 is provided to correct thepixel brightness of the saturated pixel(s) based on the calculatedinterpolated brightness value as further described above. Finally,although not specifically shown, an image generator is provided todisplay on a display device the captured image data having the correctedpixel brightness of the saturated at least one pixel. The imagegenerator can be decoder 141 and the display device can be display 142described above according to an exemplary embodiment.

FIG. 16 is a flowchart illustrating an example method 600 for pixelbrightness correction according to an exemplary embodiment. As shown,initially in a step 1602, process an image from the image sensor bycomparing a pixel brightness of the image to a first threshold and asecond threshold. For example, a pixel brightness measurement engine mayfirst measure a pixel brightness of a plurality of pixels for image datacaptured by a camera and a pixel saturation comparator can then comparethe measured pixel brightness of one or more of the plurality of pixelswith first and second saturation thresholds

In a step 1604, select the pixel brightness when the pixel brightness isbelow the first threshold. For example, a pixel brightness selector canselect the measured pixel brightness of the one pixel as an output pixelbrightness when the measured pixel brightness of the at least pixel isless than the first saturation threshold.

In a step 1606, select a neighboring pixel brightness when the pixel isabove the second threshold. For example, the pixel brightness selectorcan select a measured pixel brightness of at least one neighboring pixelas the output pixel brightness when the measured pixel brightness of theat least pixel is greater than the second saturation threshold; and

In a step 1608, perform an average of pixel brightness when the pixelbrightness is between the first and second threshold. For example, inone aspect, a pixel brightness interpolator can calculate aninterpolated brightness value when the brightness is between the firstand second thresholds to correct the saturated pixel(s). Thisinterpolated brightness value can be based on the measured brightnessvalue of the saturated pixel and a brightness value of one or moreneighboring pixels of the plurality of pixels as described above.

Finally, after the pixel brightness has been corrected, an imagegenerator displays on a display device the image data having theselected output pixel brightness of the at least one pixel of at leastone of the plurality of pixels. As a result, the generated correctedimage can be displayed without artifacts due to the saturated pixel(s),which have been replaced with a pixel correction value based on a pixelbrightness one or more unsaturated (e.g., neighboring) pixels in theimage sensor.

By way of example and without limitation, the aspects of the presentdisclosure are presented with reference to systems and methods used toconfigure various components of a video production system that may beused for production of television programming or at sports events. Thevarious concepts presented throughout this disclosure may be implementedacross a broad variety of imaging applications, including systems thatcapture and process video and/or still images, video conferencingsystems and so on.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but is to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. All structural andfunctional equivalents to the elements of the various aspects describedthroughout this disclosure that are known or later come to be known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the claims. Moreover,nothing disclosed herein is intended to be dedicated to the publicregardless of whether such disclosure is explicitly recited in theclaims. No claim element is to be construed under the provisions of 35U.S.C. § 112, sixth paragraph, unless the element is expressly recitedusing the phrase “means for” or, in the case of a method claim, theelement is recited using the phrase “step for.”

What is claimed is:
 1. An image processing system for correctingbrightness of saturated pixels in an image sensor, the image processingsystem comprising: a pixel array comprising a plurality of pixels eachconfigured to capture image data; a pixel brightness measurement engineconfigured to measure a pixel brightness of the at least one pixel ofthe plurality of pixels for the captured image data; a pixel saturationdeterminer configured to determine that the at least one pixel issaturated when the measured pixel brightness is greater than apredetermined pixel saturation threshold; a pixel brightnessinterpolator configured to calculate an interpolated brightness value tocorrect the saturated at least one pixel based on the measuredbrightness value of the saturated at least one pixel and a brightnessvalue of at least one neighboring pixel of the plurality of pixels thatis adjacent to the saturated at least one pixel; a pixel brightnesscorrector configured to correct the pixel brightness of the saturated atleast one pixel based on the calculated interpolated brightness value;and an image generator configured to display on a display device thecaptured image data having the corrected pixel brightness of thesaturated at least one pixel.
 2. The image processing system accordingto claim 1, wherein the pixel saturation determiner is furtherconfigured to determine that the at least one pixel is unsaturated whenthe measured pixel brightness is less than a predetermined pixelunsaturation threshold.
 3. The image processing system according toclaim 2, wherein the pixel brightness corrector is further configured touse the measured pixel brightness of the at least one pixel withoutcorrecting the pixel brightness when the pixel saturation determinerdetermines that the at least one pixel is unsaturated.
 4. The imageprocessing system according to claim 1, wherein the pixel brightnessinterpolator is further configured to calculate the interpolatedbrightness value by averaging measured gains of the saturated at leastone pixel and the at least one neighboring pixel, respectively.
 5. Theimage processing system according to claim 1, wherein the pixelbrightness corrector is further configured to correct the pixelbrightness of the saturated at least one pixel by replacing the measuredpixel brightness value of the saturated at least one pixel with thebrightness value of the at least one neighboring pixel when the measuredpixel brightness is greater than a predetermined high pixel saturationthreshold that is greater than the predetermined pixel saturationthreshold.
 6. The image processing system according to claim 5, whereinthe pixel brightness corrector is further configured to replace themeasured pixel brightness value of the saturated at least one pixel withaverage brightness values of a plurality of neighboring pixels.
 7. Theimage processing system according to claim 1, wherein the pixelbrightness interpolator is further configured to calculate theinterpolated brightness value by performing a weighted average of themeasured brightness value of the saturated at least one pixel and thebrightness value of at least one neighboring pixel.
 8. An imageprocessing system for correcting brightness of saturated pixels in animage sensor, the image processing system comprising: a pixel brightnessmeasurement engine configured to measure a pixel brightness of aplurality of pixels for image data captured by a camera; a pixelsaturation comparator configured to compare a measured pixel brightnessof at least one of the plurality of pixels with first and secondsaturation thresholds; a pixel brightness selector configured to selectthe measured pixel brightness of the at least one pixel as an outputpixel brightness when the measured pixel brightness of the at leastpixel is less than the first saturation threshold, and to select ameasured pixel brightness of at least one neighboring pixel as theoutput pixel brightness when the measured pixel brightness of the atleast pixel is greater than the second saturation threshold; and animage generator configured to display on a display device the image datahaving the selected output pixel brightness of the at least one pixel ofat least one of the plurality of pixels.
 9. The image processing systemaccording to claim 8, further comprising a pixel array comprising theplurality of pixels each configured to capture the image data.
 10. Theimage processing system according to claim 8, further comprising a pixelbrightness interpolator configured to calculate an interpolatedbrightness value when the measured pixel brightness of at least onepixel is between the first and second saturation thresholds.
 11. Theimage processing system according to claim 10, wherein the pixelbrightness interpolator is further configured to calculate theinterpolated brightness value based on the measured brightness value ofthe at least one pixel and a brightness value of at least oneneighboring pixel of the plurality of pixels that is adjacent to thesaturated at least one pixel.
 12. The image processing system accordingto claim 11, wherein the pixel brightness interpolator is furtherconfigured to calculate the interpolated brightness value by averagingmeasured gains of the saturated at least one pixel and the at least oneneighboring pixel, respectively.
 13. The image processing systemaccording to claim 10, wherein the pixel brightness interpolator isfurther configured to calculate the interpolated brightness value byperforming a weighted average of the measured brightness value of thesaturated at least one pixel and the brightness value of at least oneneighboring pixel.
 14. An image processor for correcting brightness ofsaturated pixels of a captured image, the image processor comprising: apixel saturation determiner configured to determine whether at least onepixel in an image sensor is saturated when a pixel brightness of the atleast one pixel is greater than at least one saturation threshold; andan image enhancer configured to generate a corrected image without oneor more image artifacts due to the saturated at least one pixel byreplacing the pixel brightness of the saturated at least one pixel witha pixel correction value based on a pixel brightness of at least oneunsaturated pixel in the image sensor.
 15. The image processor accordingto claim 14, further comprising an image generator configured to displayon a display device the corrected image without the at least oneartifact.
 16. The image processor according to claim 14, furthercomprising a pixel brightness measurement engine configured to measurethe pixel brightness of at least one pixel in the image sensor.
 17. Theimage processor according to claim 16, further comprising a pixelbrightness selector configured to select the measured pixel brightnessof the at least one pixel as the pixel correction value when themeasured pixel brightness of the at least pixel is less than the atleast one saturation threshold.
 18. The image processor according toclaim 17, wherein the pixel brightness selector is further configured toselect a measured pixel brightness of at least one neighboring pixel asthe pixel correction value when the measured pixel brightness of the atleast pixel is greater than the at least one saturation threshold. 19.The image processor according to claim 14, further comprising a pixelbrightness interpolator configured to calculate the pixel correctionvalue to correct the saturated at least one pixel based on the measuredbrightness value of the saturated at least one pixel and a brightnessvalue of at least one neighboring unsaturated pixel of the plurality ofpixels that is adjacent to the saturated at least one pixel.
 20. Theimage processor according to claim 19, wherein the pixel brightnessinterpolator is further configured to calculate the pixel correctionvalue by averaging measured gains of the saturated at least one pixeland the at least one neighboring unsaturated pixel, respectively. 21.The image processor according to claim 20, wherein the image enhancer isfurther configured to correct the pixel brightness of the saturated atleast one pixel by replacing the measured pixel brightness value of thesaturated at least one pixel with the brightness value of the at leastone neighboring adjacent pixel when the measured pixel brightness isgreater than a predetermined high pixel saturation threshold that isgreater than the at least one saturation threshold.
 22. The imageprocessor according to claim 19, wherein the pixel brightnessinterpolator is further configured to calculate the pixel correctionvalue by average a plurality of brightness values of a neighboringunsaturated pixels.
 23. The image processor according to claim 19,wherein the pixel brightness interpolator is further configured tocalculate the pixel correction value by performing a weighted average ofthe measured brightness value of the saturated at least one pixel andthe brightness value of at least one neighboring unsaturated pixel.